{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Evaluating Object Detections with FiftyOne\n",
    "\n",
    "This walkthrough demonstrates how to use FiftyOne to perform hands-on evaluation of your detection model.\n",
    "\n",
    "It covers the following concepts:\n",
    "\n",
    "-   Loading a dataset with ground truth detections\n",
    "-   Adding model predictions to your dataset\n",
    "-   Performing COCO-style evaluation of the predictions\n",
    "-   Sorting and searching samples by model performance\n",
    "-   Performing complex queries on your dataset and visualizing label efficacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "In this tutorial, we'll use an off-the-shelf [Faster R-CNN detection model](https://pytorch.org/docs/stable/torchvision/models.html#faster-r-cnn) provided by PyTorch. To use it, you'll need to install `torch` and `torchvision`, if necessary.\n",
    "\n",
    "We'll also need `pycocotools` to download the COCO dataset from the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "# Modify as necessary (e.g., GPU install). See https://pytorch.org for options\n",
    "!pip install torch\n",
    "!pip install torchvision\n",
    "\n",
    "!pip install pycocotools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following snippet will download the pretrained model from the web and load it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model ready\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "\n",
    "# Run the model on GPU if it is available\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Load a pre-trained Faster R-CNN model\n",
    "model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)\n",
    "model.to(device)\n",
    "_ = model.eval()\n",
    "\n",
    "print(\"Model ready\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll perform our analysis on the validation split of the [COCO dataset](https://cocodataset.org/#home), which is conveniently available for download via the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_creation/zoo.html).\n",
    "\n",
    "The snippet below will download the validation split to `~/fiftyone/coco-2017/validation`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Split 'validation' already downloaded\n",
      "Loading 'coco-2017' split 'validation'\n",
      " 100% |█████| 5000/5000 [32.1s elapsed, 0s remaining, 158.8 samples/s]\n"
     ]
    }
   ],
   "source": [
    "import fiftyone.zoo as foz\n",
    "\n",
    "dataset = foz.load_zoo_dataset(\"coco-2017\", split=\"validation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the dataset to see what we downloaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:           coco-2017-validation\n",
      "Num samples:    5000\n",
      "Persistent:     False\n",
      "Info:           {'classes': ['0', 'person', 'bicycle', ...]}\n",
      "Tags:           ['validation']\n",
      "Sample fields:\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n"
     ]
    }
   ],
   "source": [
    "# Print some information about the dataset\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Detection: {\n",
      "    'id': '5f46a8e098ce209ec9da498f',\n",
      "    'label': 'potted plant',\n",
      "    'bounding_box': BaseList([0.37028125, 0.33453052, 0.03859375, 0.16314554]),\n",
      "    'confidence': None,\n",
      "    'attributes': BaseDict({\n",
      "        'area': <NumericAttribute: {'value': 531.8071000000001}>,\n",
      "        'iscrowd': <NumericAttribute: {'value': 0.0}>,\n",
      "    }),\n",
      "}>\n"
     ]
    }
   ],
   "source": [
    "# Print a sample ground truth detection\n",
    "sample = dataset.first()\n",
    "print(sample.ground_truth.detections[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the ground truth detections are stored in the `ground_truth` field of the samples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add predictions to dataset\n",
    "\n",
    "Now let's generate some predictions to analyze.\n",
    "\n",
    "The code below performs inference with the Faster R-CNN model on a randomly chosen subset of 100 samples from the dataset and stores the resulting predictions in a `faster_rcnn` field of the samples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose a random subset of 100 samples to add predictions to\n",
    "predictions_view = dataset.take(100, seed=51)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 100% |█████| 100/100 [133.2ms elapsed, 13.2s remaining, 7.5 samples/s]\n",
      "Finished adding predictions\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import os\n",
    "from PIL import Image\n",
    "\n",
    "from torchvision.transforms import functional as func\n",
    "\n",
    "import fiftyone as fo\n",
    "\n",
    "\n",
    "# Get class list\n",
    "classes = dataset.info[\"classes\"]\n",
    "\n",
    "# Add predictions to samples\n",
    "with fo.ProgressBar() as pb:\n",
    "    for sample in pb(predictions_view):\n",
    "        # Load image\n",
    "        image = Image.open(sample.filepath)\n",
    "        image = func.to_tensor(image).to(device)\n",
    "        c, h, w = image.shape\n",
    "        \n",
    "        # Perform inference\n",
    "        preds = model([image])[0]\n",
    "        labels = preds[\"labels\"].cpu().detach().numpy()\n",
    "        scores = preds[\"scores\"].cpu().detach().numpy()\n",
    "        boxes = preds[\"boxes\"].cpu().detach().numpy()\n",
    "        \n",
    "        # Convert detections to FiftyOne format\n",
    "        detections = []\n",
    "        for label, score, box in zip(labels, scores, boxes):\n",
    "            # Convert to [top-left-x, top-left-y, width, height]\n",
    "            # in relative coordinates in [0, 1] x [0, 1]\n",
    "            x1, y1, x2, y2 = box\n",
    "            rel_box = [x1 / w, y1 / h, (x2 - x1) / w, (y2 - y1) / h]\n",
    "\n",
    "            detections.append(fo.Detection(\n",
    "                label=classes[label],\n",
    "                bounding_box=rel_box,\n",
    "                confidence=score\n",
    "            ))\n",
    "        \n",
    "        # Save predictions to dataset\n",
    "        sample[\"faster_rcnn\"] = fo.Detections(detections=detections)\n",
    "        sample.save()\n",
    "\n",
    "print(\"Finished adding predictions\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate detections\n",
    "\n",
    "Now that we have samples with ground truth and predicted objects, let's use FiftyOne to evaluate the quality of the detections.\n",
    "\n",
    "We'll start with some basic analysis of the predictions according to their confidence scores; then we'll compute true/false positives for each sample and analyze those."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying a confidence threshold\n",
    "\n",
    "FiftyOne provides the ability to [write expressions](https://voxel51.com/docs/fiftyone/user_guide/using_views.html#filtering) that match, filter, and sort detections based on their attributes. See [using DatasetViews](https://voxel51.com/docs/fiftyone/user_guide/using_views.html) for full details.\n",
    "\n",
    "For example, let's generate a view that contains only detections whose `confidence` is at least `0.75`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fiftyone import ViewField as F\n",
    "\n",
    "# Only keep detections with confidence >= 0.75\n",
    "high_conf_view = predictions_view.filter_detections(\"faster_rcnn\", F(\"confidence\") > 0.75)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset:        coco-2017-validation\n",
      "Num samples:    100\n",
      "Tags:           ['validation']\n",
      "Sample fields:\n",
      "    filepath:     fiftyone.core.fields.StringField\n",
      "    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n",
      "    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    faster_rcnn:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "Pipeline stages:\n",
      "    1. Take(size=100, seed=51)\n",
      "    2. FilterDetections(field='faster_rcnn', filter={'$gt': ['$$this.confidence', 0.75]})\n"
     ]
    }
   ],
   "source": [
    "# Print some information about the view\n",
    "print(high_conf_view)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Detection: {\n",
      "    'id': '5f46a90698ce209ec9db7c8b',\n",
      "    'label': 'dog',\n",
      "    'bounding_box': BaseList([0.38723122, 0.54480510, 0.56090901, 0.40119233]),\n",
      "    'confidence': 0.9992108345031738,\n",
      "    'attributes': BaseDict({}),\n",
      "}>\n"
     ]
    }
   ],
   "source": [
    "# Print a sample prediction from the view\n",
    "sample = high_conf_view.first()\n",
    "print(sample.faster_rcnn.detections[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we want to study these high confidence detections (those with `confidence >= 0.75`) in more detail.\n",
    "\n",
    "We can conveniently do that by creating a new field on our samples that contains only the detections from the filtered \n",
    "view:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 100% |█████| 100/100 [4.3s elapsed, 0s remaining, 32.6 samples/s]      \n",
      "Name:           coco-2017-validation\n",
      "Num samples:    5000\n",
      "Persistent:     False\n",
      "Info:           {'classes': ['0', 'person', 'bicycle', ...]}\n",
      "Tags:           ['validation']\n",
      "Sample fields:\n",
      "    filepath:       fiftyone.core.fields.StringField\n",
      "    tags:           fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:       fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n",
      "    ground_truth:   fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    faster_rcnn:    fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    faster_rcnn_75: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n"
     ]
    }
   ],
   "source": [
    "# Create a new `faster_rcnn_75` field on `dataset` that contains the detections\n",
    "# from the `faster_rcnn` field of the samples in `high_conf_view`\n",
    "new_field = \"faster_rcnn_75\"\n",
    "dataset.clone_field(\"faster_rcnn\", new_field, samples=high_conf_view)\n",
    "\n",
    "# Verify that the new field was created\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform COCO-style evaluation\n",
    "\n",
    "Now let's evaluate the quality of the detections in the `faster_rcnn_75` field of our samples with respect to the ground truth detections in the `ground_truth` field.\n",
    "\n",
    "FiftyOne provides a `fiftyone.utils.eval` module that contains a collection of utility methods for performing evaluation of model predictions.\n",
    "\n",
    "In particular, we'll use the [evaluate_detections()](https://voxel51.com/docs/fiftyone/api/fiftyone.utils.eval.coco.html#fiftyone.utils.eval.coco.evaluate_detections) method, which performs the following for each sample:\n",
    "\n",
    "- Matches predicted and ground truth detections across a range of [Intersection over Union (IoU)](https://en.wikipedia.org/wiki/Jaccard_index) values: `[0.50, 0.55, ..., 0.90, 0.95]`\n",
    "\n",
    "- Computes true positive (TP), false positive (FP), and false negative (FN) counts for each sample\n",
    "\n",
    "- Stores this information in the dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating detections...\n",
      " 100% |███| 100/100 [4.6s elapsed, 0s remaining, 27.5 samples/s]      \n"
     ]
    }
   ],
   "source": [
    "import fiftyone.utils.eval as foue\n",
    "\n",
    "foue.evaluate_detections(predictions_view, \"faster_rcnn_75\", gt_field=\"ground_truth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the view to see what information was added:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset:        coco-2017-validation\n",
      "Num samples:    100\n",
      "Tags:           ['validation']\n",
      "Sample fields:\n",
      "    filepath:       fiftyone.core.fields.StringField\n",
      "    tags:           fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)\n",
      "    metadata:       fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)\n",
      "    ground_truth:   fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    faster_rcnn:    fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    faster_rcnn_75: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)\n",
      "    tp_iou_0_75:    fiftyone.core.fields.IntField\n",
      "    fp_iou_0_75:    fiftyone.core.fields.IntField\n",
      "    fn_iou_0_75:    fiftyone.core.fields.IntField\n",
      "Pipeline stages:\n",
      "    1. Take(size=100, seed=51)\n"
     ]
    }
   ],
   "source": [
    "print(predictions_view)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each sample now contains new fields `tp_iou_0_75`, `fp_iou_0_75`, and `fn_iou_0_75` corresponding to the total true positive (TP), false positive (FP), and false negative (FN) counts for the detections in the `faster_rcnn_75` field of the samples at an IoU of 0.75 (this value can be customized via the `save_iou` argument of the evaluation method).\n",
    "\n",
    "In addition, the predictions in the `faster_rcnn_75` field of each sample contain a new `ground_truth_eval` field that tabulates TP, FP, and FN counts for each IoU under test.\n",
    "\n",
    "Finally, the individual detections in the `faster_rcnn_75` field of each sample have a new `ground_truth_eval` field that contains:\n",
    "\n",
    "- An `eval_id` field that specifies a UUID for the detection\n",
    "\n",
    "- An `ious` field that contains the IoUs for every class of that detection with respect to the ground truth detections of that class\n",
    "\n",
    "- A `matches` field that lists the `eval_id` and `iou` for the matching ground truth detection (if any) using the matching algorithm provided by `pycocotools`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize detections\n",
    "\n",
    "Now let's visualize this evaluation information that we've added to our dataset!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing bounding boxes\n",
    "\n",
    "First, let's launch the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html) and view the ground truth and predicted bounding boxes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "App launched\n"
     ]
    }
   ],
   "source": [
    "# Open the dataset in the App\n",
    "session = fo.launch_app(dataset=dataset)\n",
    "\n",
    "# Load view containing the subset of samples for which we added predictions\n",
    "session.view = predictions_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![launch](images/eval_dets/eval_dets_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each field of the samples are shown as togglable bubbles on the left sidebar which can be used to control whether ground truth detections (`ground_truth`), raw predictions (`faster_rcnn`), or high confidence predictions (`faster_rcnn_75`) are rendered on the images:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![bubbles](images/eval_dets/eval_dets_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manually select samples of interest\n",
    "\n",
    "You can select images in the App by clicking on them. Then, you can create a view that contains only those samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The currently selected images in the App\n",
    "selected_samples = session.selected\n",
    "\n",
    "# Create a new view that contains only the selected samples\n",
    "# And open this view in the App!\n",
    "session.view = dataset.select(selected_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![selected](images/eval_dets/eval_dets_3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's reset the session to show the entire dataset again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resets the session; the entire dataset will now be shown\n",
    "session.view = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View the best-performing samples\n",
    "\n",
    "Recall that a `tp_iou_0_75` field was added to each sample that tabulates the number of true positive detections in that sample.\n",
    "\n",
    "Let's create a view that sorts by `tp_iou_0_75` so we can see the best-performing cases of our model (i.e., the samples with the most correct predictions):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show samples with most true positives first\n",
    "session.view = predictions_view.sort_by(\"tp_iou_0_75\", reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![tprev](images/eval_dets/eval_dets_4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View the worst-performing samples\n",
    "\n",
    "Similarly, we can sort by the `fp_iou_0_75` field to see the worst-performing cases of our model (i.e., the samples with the most false positive predictions):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show samples with most false positives first\n",
    "session.view = predictions_view.sort_by(\"fp_iou_0_75\", reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![fprev](images/eval_dets/eval_dets_5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering by bounding box area\n",
    "\n",
    "`DatasetView` pipelines are extremely powerful. For example, let's look at how our model performed on small objects by creating a view that contains only detections whose bounding box area is less than `0.005`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bounding box format is [top-left-x, top-left-y, width, height]\n",
    "bbox_area = F(\"bounding_box\")[2] * F(\"bounding_box\")[3]\n",
    "\n",
    "# Create a view that contains only predictions whose area is < 0.005\n",
    "small_boxes_view = predictions_view.filter_detections(\"faster_rcnn_75\", bbox_area < 0.005)\n",
    "\n",
    "session.view = small_boxes_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![small](images/eval_dets/eval_dets_6.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing detections in a crowd\n",
    "\n",
    "Recall that our ground annotations from the COCO dataset have an `iscrowd = 0/1` attribute that indicates whether a box contains multiple instances of the same object.\n",
    "\n",
    "Let's create a view that contains only samples with at least one detection for which `iscrowd` is 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a view that contains only samples for which at least one detection has \n",
    "# its iscrowd attribute set to 1\n",
    "crowded_images_view = predictions_view.match(\n",
    "    F(\"ground_truth.detections\").filter(F(\"attributes.iscrowd.value\") == 1).length() > 0\n",
    ")\n",
    "\n",
    "session.view = crowded_images_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![crowd](images/eval_dets/eval_dets_7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More complex insights\n",
    "\n",
    "Let's combine our previous operations to form more complex queries that provide deeper insight into the quality of our detections.\n",
    "\n",
    "For example, let's sort our view of crowded images from the previous section in decreasing order of false positive counts, so that we can see samples that have many (allegedly) spurious predictions in images that are known to contain crowds of objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_crowded_images_view = crowded_images_view.sort_by(\n",
    "    \"fp_iou_0_75\", reverse=True\n",
    ")\n",
    "\n",
    "session.view = sorted_crowded_images_view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![crowdsort](images/eval_dets/eval_dets_8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the above view to another view that just sorts by false positive count, regardless of whether the image is crowded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.view = predictions_view.sort_by(\"fp_iou_0_75\", reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![fprev2](images/eval_dets/eval_dets_5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See anything interesting?\n",
    "\n",
    "Comparing the individual examples, we see that the samples that contain many false positives are the ones where the \n",
    "underlying ground truth bounding box was missing the `iscrowd` attribute! The effect of this omission is that crowds of correct predictions are evaluated as false positives even though they are true positives.\n",
    "\n",
    "In other words, the quality of model may not be responsible for the purportedly low quantitative performance of the detections; in fact, the ground truth annotations from the COCO dataset should be refined to fix the missing `iscrowd` annotations!\n",
    "\n",
    "This conclusion would have been nearly impossible to achieve without visually inspecting the individual samples in the dataset according to the variety of criteria that we considered in this tutorial.\n",
    "\n",
    "FiftyOne enables rapid experimentation with your datasets!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
